Field of the Art
The disclosure relates to the field of digital media, and more particularly to the field of synchronized digital multimedia playback.
Discussion of the State of the Art
Today there are many forms of digital media, many types of digital media sources, many types of digital media playback (rendering) systems and lots of ways of connecting media sources to media playback systems.
Digital media, hereafter referred to as media, comes in many forms, formats and containers, including Digital Video Disks, media files and media streams. The media contents can be audio, video, images or meta data media components and various combinations of each. For example, a popular audio format is known as MP3 and a popular video format is H264. MP3 is an audio-specific media format that was designed by the Moving Picture Experts Group (MPEG) as part of its MPEG-1 standard and later extended in the MPEG-2 standard. H264 is a standard developed by the International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) joint working group, the Moving Picture Experts Group (MPEG). Movies are typically multimedia formats with a video and multiple audio channels in it. For example, a 5.1 movie contains 1 video channel (media component) and 6 audio channels (audio components). 5.1 is the common name for six channel surround sound multichannel audio systems.
Digital media sources include media devices such as Digital Video Disk players, Blu-ray players, computer and mobile devices, and internet based “cloud” media services. Blu-ray Disc (BD) is an optical disc storage medium developed by the Blu-ray Disc Association. Internet based media services include services such as NETFLIX™ and SPOTIFY™. Netflix is a media service and trademark of Netflix Inc. SPOTIFY™ is a media service and trademark of Spotify Ltd. Digital media playback (media rendering destinations) systems include computer based devices, laptops and smartphones, as well as network audio and video devices. A SmartTV is an example of a digital media-rendering device that can play media from an internet (cloud) based media service such as NETFLIX™. A SmartTV, which is also sometimes referred to as “Connected TV” or “Hybrid TV”, is used to describe the integration of the internet and Web features into modern television sets and set-top boxes, as well as the technological convergence between computers and these television sets/set-top boxes. An Internet radio device is another example of a digital media rendering device.
The connectivity between these media sources and devices is varied, but is evolving over time towards network-based connectivity using Internet protocol (IP) protocols. This is because IP connectivity is convenient, ubiquitous, and cheap.
IP networks come in many forms; the most prevalent being Ethernet based wired IP networking. Ethernet is a family of computer networking technologies for local area networks (LANs) that is standardized as IEEE (Institute of Electrical and Electronics Engineers) Standard 802.3. In recent years with the prevalence of mobile computing devices, Wi-Fi (a type of IP network) has become the most popular means for connecting network devices wirelessly. Wi-Fi is a trademark of the Wi-Fi Alliance and a brand name for products using the IEEE 802.11 family of standards. IP networks can use several different types of messaging including unicast, multicast, and broadcast messaging, such messaging being the sending of IP packets.
The term “Unicast” may be used to refer to a type of Internet Protocol transmission in which information is sent from only one sender to only one receiver. In other words, unicast transmission is a one-to-one node transmission between two nodes only. In unicasting each outgoing packet has a unicast destination address, which means it is destined for a particular destination that has that address. All other destinations that may hear that packet ignore the packet, if the packet's destination address is not the same as that destination's address.
Many IP protocols are accessed from software programs via a socket application programming interface. This socket interface is defined as part of the POSIX standard. POSIX is an acronym for “Portable Operating System Interface”, which is a family of standards specified by the IEEE for maintaining compatibility between operating systems.
The convenience and benefits of IP networking means that all of these media sources and playback systems, if not already network enabled, are becoming network enabled. Many Blu-ray players now have Ethernet and Wi-Fi network connectivity. Today most higher-end TVs are smart TVs that have network capability. Similarly, audio play back devices and even radios are network and Internet enabled.
Mobile devices, such as mobile phones, tablets, document readers, or notebooks, are able to receive and store media and have powerful multimedia (audio and video) capabilities and may be connected to the internet via cell phone data services or broadband links (such as Wi-Fi) that are high bandwidth and can access online media services that have wide and deep content.
The use cases or applications of these various forms of digital media, media services and media sources and playback systems have been evolving. Initially it was enough to connect a media source to a media destination over an IP network. This is widely used today with Internet based media source services, such as Netflix and a computer as a media destination. Users watch Netflix movies streamed over a wired IP network (the internet) to a computer. This is a case of a single point (one IP source) to single point (one IP destination) connection over a wired IP network. Even though the Netflix media service may send the same media to multiple households, each of these is a single point to single point TCP/IP connection. A further evolution of this is to use a wireless, Wi-Fi connection, instead of a wired Ethernet connection. This is still a single point to single point connection.
A further extension of the above use cases exists, where a media source connects to multiple destinations rather than a single destination. These are single point (one IP source) to multi point (multiple IP destinations) applications. An example would be where a user is playing a 5.1 movie media file to a wireless video playback device and 6 independent wireless audio destinations making up a full 5.1 surround sound system. In this case the media is going from one media source to 7 media destinations simultaneously. In another example, a user is playing music from one media source to six audio playback systems placed around the home in six different rooms.
In both of these cases, it is necessary to play (render) the media at all destinations time synchronously. Furthermore, it is necessary to limit the use of resources at the media source, such as keeping memory use to a minimum. In addition, it is necessary with multiple devices receiving media to manage network bandwidth efficiently.
Currently when the same media data needs to be sent to multiple network destinations, the general technique for doing so is to use data multicasting to the multiple destinations that need to receive the data. In such a system, the media is multicast to all the destinations and it is up to each destination to attempt to render the media appropriately. If during rendering there is an error where a renderer does not receive new media data or does not receive it correctly, the renderer may render erroneous data and then attempt to recover and continue correct media rendering from the point after the error when correct data is received.
In the applications envisioned here, there is a need to send media from a source to multiple media devices, such as TV and speakers in the same listening and viewing space. Furthermore, there is a need to send this media over a wireless network such as Wi-Fi.
For these applications, this means all of the media rendering devices, such as speakers, that are in the same listening or viewing zone, need to be precisely synchronized to each other, so the listener and/or viewer does not discern any unintended media experience.
Secondly, because the media is transported over wireless, there is a very high likely hood of a media error, where the media is not received at each destination reliably or uniformly. If using broadcast or multicasts to send packets, the same broadcast or multicast packet, may be received at one destination but not received/heard by another destination.